First Hit Fwd Refs 
End of Result Set 



Previous Doc 



Next Doc 



Go to Doc# 




Lll: Entry 13 of 13 



File: USPT 



Dec 19, 



1995 



DOCUMENT- IDENTIFIER: US 5477459 A 

TITLE: Real time three-dimensional machine locating system 
Abstract Text (1) : 

A method and system for locating the point P.sub.O which defines the location of a 
machine, e.g. an earth moving machine, a surveying or locating machine, etc., in 
relation to a tract or area of interest by measuring the angles defined by two 
different sets of reference points selected from at least three reference points 
that are in known relation to the tract or area. 

Brief Summary Text (2) : 

In the field of surveying and distance measuring equipment, there is a need to 
determine the position of objects (markers, etc. ) relative to fixed points. There 
are several methods for doing this. In general, two approaches are used. One method 
involves the measurement of linear distances; another, the measurement of angles 
and distances. Usually a construction site is surveyed and the topography is 
plotted before work begins. Fixed flags tell the equipment operators to either cut 
or fill a particular location. An iterative process of survey and grade is used to 
work the site into the final topography prior to actual construction. This 
iterative process is slow and laborious and therefore expensive. Since there is a 
great deal of human work involved, it is also error prone. 

Brief Summary Text (17) : 

U.S. Pat. No. 4,774,403, to Arts, on Sep. 27, 1988, discloses a position measuring 
device using a triangulation scheme for position sensing. A laser diode projects a 
spot upon a target surface. The image of the spot is focused on a charge coupled 
device linear image sensor. The image of the sensor is indicative of the distance 
from the device to the target surface. 

Brief Summary Text (18) : 

U.S. Pat. No. 5,032,023, to Schneiter, on Jul. 16, 1991, discloses an apparatus for 
a variable depth triangulation ranging system, the ranging system including means 
for emitting a fight beam, means for focusing an emitted light beam on an object, 
and lens means for imaging reflected light onto an input end of said apparatus. The 
apparatus includes an optical fiber bundle, the bundle including an input end and 
an output end, the bundle input end being disposed so as to intercept transmissions 
from said lens means, means for scanning, in a time-based manner, the bundle output 
end, the scanning means being configured to generate a time-based signal indicative 
of a light signals present at the bundle output end, and means for calculating 
system geometry and range from the time-based signal. 

Brief Summary Text (19) : 

Tanabe, Toru, Experimental Validation of GPS-INS-STAR Hybrid Navigation System for 
Space Autonomy, International Astronautical Congress, 39th, Bangalore, India, Oct. 
8-15, 1988. 12 p. (Abst.) describes the gps-ins-star hybrid navigation system 
concept combines these three sensors: a ins-star hybrid system, a gps-ins system, 
and a gps-ins-star system. It was concluded that a star field simulator had been 
developed to serve as a source of radiation for the astro star tracker. A minimum 



of three simulated guide stars with variable magnitudes were needed to fully test 
the star tracker performance under simulated mission conditions. 



Brief Summary Text (23) : 

Van der Grinten, H. F., Navigation, vol. 22, Summer 1975, p. 128-134, (Abst) , 
cataloged forty-six celestial lines of position obtained aboard a ship at sea are 
compared with the position as interpolated between NAVSAT fixes. Least squares 
adjustment of the combined celestial lines of position yielded a fix with an 
essentially circular error ellipse. The directional distribution of stars was well 
balanced. The navsat positions and the combined celestial fix were found to be 
substantially in agreement. Using the NAVSAT positions as a reference the mean 
error of well balanced celestial fixes as derived from the data and the method of 
least squares can be expected to be less than 0.8 nm for a 3-star fix and 0.1 nm 
for a 46-star fix. These results permit the construction of probability contours 
around well balanced celestial fixes consisting of various numbers of observations. 

Brief Summary Text (24): 

Wertz, J. R., Spaceflight, 14; June 1972, p. 206-216, (Abst) asserted that the most 
essential requirement is an atlas of the three-dimensional positions of all of the 
stars in the region to be explored and any bright navigational stars that might lie 
outside this region. In addition to accurate stellar positions, data about the 
velocities of the listed stars are required together with some means of accurately 
identifying individual stars — specifically, their absolute magnitudes and detailed 
spectra. Given the interstellar navigation atlas, there are many ways for the 
navigator to determine his position and velocity. Effects of high speed motion are 
discussed, together with the possibility of visual disorientation due to a 
displacement of the star field caused by small shiets of location. 

Brief Summary Text (25) : 

A method for determining the position of an object in a 3D space has been proposed. 
An M-array LED pattern is projected onto the object through a lens and the 
projected array is observed with a CCD camera . Each element of the M— array is time 
modulated with an M-sequence having different phase. The cross-correlation function 
between the observed signal and the reference gives information about the phase, 
and thus the point-to-point correspondence. Then simple mathematical relations are 
used to determine the exact 3D position of the object . The proposed method is 
robust to noise and highly efficient. (Measurement of 3-dimensional position by use 
of M-array projection, Jing-Min Sham Kashiwagi, H.; Sakata, M., dournak 
Transactions of the Society of Instrument and Control Engineers, vol.25, no. 4 
p. 389-95, 1989). 

Brief Summary Text (26) : 

The monitoring of the motion of an arbitrary mandibular point has been accomplished 
using a recording system that includes three cameras . Each camera contains a 
cylindrical lens and a linear optosensor (CCD) . Two extra-oral lightweight target 
flames (2 g) containing each three light emitting diodes are fixed on the patient's 
teeth. The system fires each LED cyclically and computes their three-dimensional 
coordinates in a reconstruction unit. From these coordinates another processing 
unit computes the three-dimensional coordinates of an arbitrary jaw point in a head 
fixed coordinate system. The computed trajectories are drawn on-line with the help 
of dedicated 3D viewing hardware on a CRT where they can be directly rotated and 
zoomed to follow intricate motion details. (Real-time, noninvasive recording and 
three-dimensional display of the functional movements of an arbitrary mandible 
point, Mesqui, F. ; Kaeser, F . ; Fischer, P., Proceedings of the SPIE — The 
International Society for Optical Engineering vol.602 p. 77-84, 1986). 

Brief Summary Text (27) : 

An optical correlator uses a light emitting diode array, in which analog signals 
from two spaced sensors are correlated in order to locate and track a target. One 



signal is clipped and digitized and clocked through a shift register, and the other 
signal is delayed. The shift register is coupled to an LED array, element for 
element. Each shift register element modulates its corresponding LED element. The 
delayed signal is also connected at its output to the LED array via a transistor, 
and modulates each LED element. Therefore, the LED elements emit light in 
proportion to the product of the two signals. The emitted light is focused onto a 
CCD imaging array where it is integrated over a period of time before being sent to 
an integrator and output display device. The output is the correlation function 
versus the time delay between the two signals. Successive outputs display the 
movement of targets. Circuit design mitigates inherent errors within the system. 
(Acoustic Signal Optical Correlator Using a Light Emitting Diode Array, Fogarty, E. 
J. U.S. Pat. No. 4,805,158, Feb. 14, 1989). 

Brief Summary Text (29) : 

A helmet mounted sight system using CCD technology is described which can be used 
outside the restricted viewing area of the head up display, but with reduced 
accuracy. It is noted that the system consists of two pans: the helmet mounted 
display and the helmet optical position sensor. It is reported that a limited 
amount of essential information (i.e. aiming reticule, weapon lock, etc.) can be 
displayed on the surface of the helmet mounted display, noting that the limit is 
set by the pilot f s inability to assimilate further data. Attention is given to the 
helmet optical position sensor which consists of two triads of LED's mounted on the 
sides of the helmet and two CCD cameras mounted in the cockpit. Finally, it is 
concluded that using the CCD to its full capability enables a small compact system 
to be produced without the need for fast analog to digital converters and a large 
digital store. (A helmet-mounted sight using CCD. technology, STEPHENSON, M.D., 
Radio and Electronic Engineer, vol. 49, Oct. 1979, p. 511-513, 1979). 

Brief Summary Text (32) : 

U.S. Pat. No. 4,994,907, to Allen, on Feb. 19, 1991, describes a color sensing CCD 
with staggered photosites, e.g., a line sensor with photosites accurately located 
for color scanning. The sensor includes a plurality of photosites arranged in a 
two-dimensional, staggered pattern which is repeated across the length of the 
sensor. Only one photosite is located for every direction perpendicular to the axis 
of the line image . Individual color filters extend over all the photosites located 
at the same perpendicular distance from the line image axis. In one embodiment, the 
filters are disposed on separate transparent members which are aligned and 
assembled over the photosites. 

Brief Summary Text (33) : 

Mollow, P. A., et al, Journal: Proceedings of the SPIE — The International Society 
for Optical Engineering, Vol.1347 p. 123-30, 1990 (Abstr.) describes a system using 
one-dimensional optical devices to perform the desired two-dimensional correlation. 
The two-dimensional correlation is performed as a series of multichannel time- 
integrating correlations between each input image line and a reference template 
that is stored in an electronic memory. The rows of the reference template are 
introduced into the processor in parallel using a one-dimensional laser diode 
array. The correlation in the vertical direction is performed using a modified 
charge-coupled device operating in the shif t-and-add mode. Key features of the 
system include the random access template memory, the custom laser diode array 
consisting of 64 individually addressed laser diodes. 

Brief Summary Text (34): 

The basic mathematical formulation of a general solution to the extraction of 
three-dimensional information from images and camera calibration has been 
described. Standard photogrammetric algorithms for the least squares estimation of 
relevant parameters are outlined together with terms and principal aspects of 
calibration and quality assessment. Three-dimensional testfields and independently 
determined reference coordinates were used for quality assessment using off the 
shelf equipment to an accuracy of 1/20/ sup th/and 1/50/ sup th/ of the pixel 



spacing in row and column direction respectively has been achieved. The system was 
used to calibrate the vision system of a ping-pong playing high-speed robot 
tracking table-tennis balls with a 50 Hz rate. Calibration of CCD -cameras for 
machine vision and robotics, Beyer, H. A., Journal: Proceedings of the SPIE — The 
International Society for Optical Engineering vol. 1197 p. 88-98, 1990 (Abstr) . Two 
sensors were used in a system for autonomous rendezvous and docking manoeuvres in 
space. The first is a proximity sensor, which is dedicated for distance and angle 
measurements. The sensor's equipment consists of three units: the position- 
sensitive detector, the illuminator and the common optics including a c.w. laser 
rangefinder. On the target side a corresponding reflector complements the necessary 
equipment to measure distance and angle deviation from boresight direction. The 
operation wavelengths are between 790 and 850 nm. The second is a high-resolution 
CCD -camera in combination with a dedicated S/W processor, which is used for 
distances between 2 m and 100 m. On the target side, a three-dimensional symmetric 
pyramid reflector configuration is used. The camera processor transforms the 
actually depicted reflector configuration differences to the expected reflector 
configuration for boresight position into distance and position information for six 
DOF. This method can achieve faster update repetition rates ( 10 Hz) than a common 
pattern recognition technique. The camera is equipped with a 880 nm wavelength LED 
illuminator. Optical sensors for position measurements, Schroer, G. et al, Sensors 
and Actuators vol. 17, no . 3-4p . 329-38 , 1989 (Abstr). 

Brief Summary Text (35) : 

Optical correlators have been used to investigate real time pattern recognition. 1- 
D devices have been used to perform the two dimensional correlation in real time by 
using an array of light emitting diodes (LED 1 s ) to input an electronically stored 
reference image into the processor in parallel. The input scene data is introduced 
into the processor one line at a time using an acousto-optic device (AOD) . 
Multichannel time integrating correlations are performed in the row direction using 
the AOD and in the column direction using a charge coupled device (CCD) operating 
in the time delay and integrate mode. A processor that correlates a 64. times. 44 
pixel binary reference image with a 256 . times . 232 input scene at video rates was 
used. A second correlator is a space integrating Fourier transform based 
correlator. A magneto optic-device (MOD) is used at the Fourier transform plane to 
rapidly change filter functions. Stalker, K., et al, International Congress on 
Optical Science and Engineering, 24-28 Apr. 1989 (Abstr) . 

Brief Summary Text (36) : 

In another application, a new sensor that can measure the two-dimensional 
environment of a robot has been reported. This sensor consists of a laser diode and 
a charge coupled device (CCD) linear array sensor built in a simple optical imaging 
system with a rotating mirror. The measuring range of this sensor is 250 mm to 6000 
mm, and the measuring time for one direction is 1 ms to 10 ms . It can measure 
distances more than 200 points to obstacles or walls around robot per second. The 
position of the walls is recognized from these measuring points using a 
microcomputer system. Development of environment sensor for vehicle robot, 
Takamasu, Kiyoshi, Journal of the Robotics Society of Japan, Volume: 5, Issue: 3, 
Pages: 199-202, June 1987 (Abstr). 

Brief Summary Text (37): 

U.S. Pat. No. 4,970,653, to Kenue, on Nov. 11, 1990, discloses an image processing 
method that operates on an image from a CCD camera viewing a roadway scene in front 
of a vehicle to detect lane markers and determine the relationship of the vehicle 
to the lane. Obstacles in the lane near the vehicle are detected and a warning is 
given to the driver. The method uses template matching techniques or a hough 
algorithm to detect the lane markers or road edges. 

Brief Summary Text (38): 

The components and instruments necessary to carry out the present invention are 
available commercially. (Components and peripherals, buyers guide, EDN VOL. : v33 



nl5 p253(27), Jul. 21, 1988, Lasers & Optronics, v7 , n4, p76(3), April, 1988.) For 
example, Infrared Associates 1 infrared detectors and related accessories have 
applications in analytical instruments, medical thermography, thermal imagers, air- 
borne scanners, high-altitude balloon experiments, laser range finders and high- 
speed missiles. They come in single and multi-element arrays and can be packaged in 
a variety of metal or glass dewars, which can be cooled or uncooled. The matched 
preamplifier circuits can be provided in single or multi-channel configurations. 
Infrared Associates, Inc., 1000 Route 130, Cranbury, N.J. 08512. Automated Waveform 
Digitizers LeCroy Corp.'s line of high-performance automated single-channel, 
waveform digitizers provides high speed wave measurement and analysis 
instrumentation. Among the benefits the. devices provide are higher accuracy 
measurements, extended memory-length waveform recording, pre-and post-trigger 
waveform recordings, high resolution transient capture, digital signal processing 
and analysis and total programmability . LeCroy Corp., 700 Chestnut Ridge Rd., 
Chestnut Ridge, N.Y. 10977-6499. A Red-emitting diode laser is available in the 
visible-wavelength range. NEC's Model NDL 3200, operates at room temperature and 
will run more than 3,000 hours without performance degradation. Fabricated in 
indium gallium arsenide phosphide, the device emits a minimum of 3 milliwatts at 
680 nanometers. The diode has a back-facet monitor photo-diode and is offered in a 
three-pin, 9-millimeter TO can. This diode laser has rise and fall times of 400 
nanoseconds. NEC Electronics, Inc., P.O. Box 7241, Mountain View, Calif. 94039. The 
4000 series of lasers from Continental Laser Corp. feature a two-tube tandem design 
which virtually eliminates any down time, according to the manufacturer. Two large- 
diameter solid Invar rods provide a 50-inch resonator structure for stability. The 
systems are designed for OEM, industrial, medical, entertainment and scientific 
applications. Multi-line power specifications are 8 to 12 W for the argon models 
and 2.5 to 4 W for krypton. Light regulation, prism wavelength selector, remote- 
control module and etalon assembly options are available. Continental Laser Corp., 
805 E. Middlefield Rd., Mountain View, Calif. 94043. XMR Inc.'s Model 5100 excimer 
laser is designed specifically for a long lifetime of industrial use. Capable of 
generating a continuous 150 watts of average power or 200 watts for shorter 
periods, the 5100 features a magnetically switched PFN and can be incorporated into 
custom laser-based systems. XMR, Inc., 5403 Betsy Ross Dr., Santa Clara, Calif. 
95054. Air- and water-cooled argon and krypton lasers, American Laser Corp., range 
in output power from 1 to 8 watts, continuous wave, multi-time configuration. The 
units are available in argon, krypton and mixed gas and offer additional features 
such as single-line operation, TEM sub. 00, extended-life tube, tunability, UV and 
service contracts. American Laser Corp., 1832 S. 3850, W., Salt Lake City, Utah 
84104. A hard-sealed, integral-mirror, plane-polarized helium- cadmium laser, the 
Model 43001, exhibits a typical polarization ratio higher than 1000:1. Measuring 
4 . times . 4 . times . 24 inches with a full Invar resonator, this laser offers high power 
stability ( [plus-or-minus] ) 1 percent over one hour), low noise ([is less than or 
=] ) 1 percent rms) and an extended warranty period (4000 hours or one year to half 
power) . The laser is also available with random polarization or with frit-sealed, 
fused-silica Brewster windows for those users switching between 325 and 442 nm 
operation. Liconix, 1390 Borregas Ave., Sunnyvale, Calif. 94089. 

Brief Summary Text (39) : 

A digital video CCD Camera and scientific investigations is available. This camera 
features high-resolution, true random triggering, high-light gain with optional 
image intensif iers , digital and standard video outputs, digital frame store and on- 
board video monitor, digital delay and exposure-time controls, and sequence 
operation for stroboscopic photography. Applications include ballistics, detonics 
and other high-speed photography including laser and fusion research and 
spectroscopy. Marco Scientific, Inc., 1055 Synnyvale-Saratoga Rd., #8, Sunnyvale, 
Calif. 94087. 

Brief Summary Text (42) : 

Herschelian-mounted remote sensing receivers and radio-metric systems are available 
from Lambda/Ten Optics. They have Herschelian-mounted Newtonian telescopes and 



collimators. These all-reflective instruments have no central obscuration and 
locate the focal plane to the side of the collimator/telescope . Used as a 
collimator, the Lambda/Ten "NC Series" projects a collimated beam from the light 
source to infinity. As a telescope the unit images light from a distance to the 
focal plane. The series is ruggedized in a heat-treated aluminum tube, fully self- 
supporting, and mountable in any orientation. Lambda/Ten Optics, 1 Lyberty Way, 
Westford, Mass. 01886. 

Brief Summary Text (47) : 

As a system, the invention is useful for locating and defining the position of an 
object in relation to a known area or tract. A plurality of point-sources of 
electromagnetic radiation, e.g. superbright LED 1 s are located, respectively, at 
each of at least three reference points in defined relationship to each other and 
to said area or tract. An angle measuring sensor for said electromagnetic energy, 
e.g. a CCD sensor system, is positioned at an unknown location P.sub.O and in known 
relation to the object the location of which is to be defined. The sensor is 
constructed and configured to measure angles defined by lines extending through 
P.sub.O and at least two of a first set of said reference points and angles defined 
by lines extending through P.sub.O and at least two of at least one additional set 
of reference points, respectively, the respective sets of reference points being 
different from each other and being selected such that a geometric figure defined 
by P.sub.O and the first set overlaps a geometric figure defined by P.sub.O and at 
least one additional set of reference points. A computer is connected to said 
sensor for receiving data defining said measured angles and input data defining the 
relationship of said reference points to each other and to said tract or area and 
for computing data defining the location of P.sub.O in relation to said tract or 
area and means are provided to cause the computed definitional data signal derived 
from said computed data defining the location of P.sub.O to a display or control 
means. The definitional data signal my be used for controlling the movement or 
operation of the object. The system may function in a single plane or in three 
dimensions and may use three-dimensional data to define location in a single plane, 
or in three dimensional space. 

Detailed Description Text (10) : 

In the special case where the points lie on a straight line on or adjacent the 
tract or area of interest, the location of P.sub.O can be calculated as in the 
general case or as described here, making reference now to FIGS. 1, 2 and 8. The 
coordinates of points P.sub.l, P. sub. 2 and P. sub. 3 are known. These three points 
preferably but not necessarily would be at one edge or slightly outside the project 
to be worked upon and lie on the same line with each other. This arrangement is for 
simplification and to eliminate one of two solutions that is mathematically 
possible . 

Detailed Description Text (20) : 

(a) Point electromagnetic radiation sources, e.g., point-light source transmitters, 
are placed at three or more reference points that are in known, defined 
relationship to each other and to the tract or area of interest . 

Detailed Description Text (25) : 

(g) The mathematical calculations described to define the location P.sub.O relative 
to one or more reference indicia relative to the area or tract are executed, 
typically by a computer that receives data directly from the sensors, thereby 
define the location P.sub.O on or in relation to the tract or area of interest . 

Detailed Description Text (29) : 

The point-source light may send out a pulse coded sequence of light bursts which 
serves to identify the particular point to send short, intense pulses for longer 
range without large average power requirements or special equipment. The light 
transmission is preferably omnidirectional in the X and Y direction and covers 
approximately 30. degree, above and below the X-Y plane in the Z direction. In a 



preferred embodiment/ the light transmitter consists of a radial cluster of superb 
fight infrared light emitting diodes or diode lasers with lenses. The typical 
emission maximum is around 880 Nm. The transmitter, once activated, can 
continuously transmit the pulse coded information sequence. A Dalsa, Inc., CCD 
camera having a CCD sensor having 256 . times . 256 pixels and a 50 mm lens using an 
infrared filter to block light below 800 nm may be used in connection with an 
infrared superbright light emitting diode, Siemens pan No. SFH484 as the point 
source. Greater accuracy can be accomplished using, for example, a 4000 pixel array 
such as is market by Dalsa. These are, of course, merely exemplary. 

Detailed Description Text (37) : 

In a preferred embodiment, the image of the point source transmitter is focused on 
the sensitive elements of a photo detector array. The electronics scan the array 
and determine the position of the image on the detector surface. Since the image 
position on the detector corresponds to the location of a particular point, e.g. 
P.sub.l, the precise angle between such point, e.g. P.sub.l, and another point 
P. sub. 2 can be determined. The radiation sensing devices can be mounted in the 
sensing system with great exactitude relative to each other. The radiation sensing 
devices can also be mounted and position to have precisely known relationships 
between the fields of view of the respective sensors, or of a single sensor in 
different positions. A single sensor system with wide-angle sensors would be 
capable of viewing, in fields that meet or overlap, in a 360. degree, solid angle, 
with, perhaps, some minor interference from the hardware of the sensor system per 
se. This capability eliminates the need for mechanical or electro-mechanical angle 
measuring or calibrating devices. Using point sources at known locations, 
calibration can be checked or adjusted, if ever necessary using only the sensor and 
computer systems. This is very simple where the two point sources are detectable by 
a single sensor in the sensor system. 

Detailed Description Text (38) : 

Although various types of photo detectors can be used including photodiode arrays, 
vidicon tubes and others, the preferred method involves the use of charge coupled 
devices (CCD) . CCDs have precision pixels (picture element) arranged in various 
formats. The image scanning is simple to do with computer driven image analyzers. 
In addition, systems with high IR sensitivity are available. To increase specific 
signal to background signal, it may be desirable to place an IR pass filter in 
front of the detector; all other light will then be blocked. CCD image sensors such 
as manufactured by Dalsa, Inc., Waterloo, Ontario, Canada, are suitable. 
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DOCUMENT-IDENTIFIER: US 20040032493 Al 

TITLE: Method for monitoring the interior and/or exterior of a vehicle, and a 
vehicle having at least one survaillance camera 

Abstract Paragraph : 

A method for monitoring a space of a vehicle. The method includes sensing the space 
using a first sensor including at least one surveillance camera so as to produce an 
image data, performing an evaluation of the image data, selecting at least one area 
of the space using the evaluation of the image data, sensing the at least one area 
using a second alignable sensor having a restricted spatial sensing range so as to 
produce a second data, and evaluating the second data. In addition, a vehicle that 
includes a first sensor having at least one surveillance camera disposed in an 
interior of the vehicle, the first sensor having a field of view at least partially 
covering at least one of the interior and an exterior of the vehicle, and a second 
alignable sensor having a restricted spatial sensing range, an alignment of the 
second sensor being controllable using the first sensor. 

Summary of Invention Paragraph : 

[0002] The invention relates to a method for monitoring the interior and/or 
exterior of a vehicle, and to a vehicle having at least one surveillance camera in 
the vehicle interior. 

Summary of Invention Paragraph : 

[0003] Motor vehicles having a camera in the vehicle interior are known, for 
example, individual cameras with a field of view to the outside can be used to 
monitor the front, side and/or rear spaces through the window panes of the vehicle. 
Again, cameras have already been proposed for observing parts of the vehicle 
interior, for example in German Patent Application No. DE-A-198 03 158, which 
exhibits a device for optically determining the state of vigilance of the operator 
of a vehicle. 

Summary of Invention Paragraph : 

[0004] The unpublished German patent application previously applied for by the 
applicant and having the official file reference DE 101 58 415.6 discloses a method 
for optically monitoring the interior of a vehicle with at least one surveillance 
camera . In this case, the sensing of the exterior is also represented by at least 
one surveillance camera . This described mode of procedure reguires complicated 
evaluation of the image data, the inf ormativeness of the evaluated data not always 
being sufficient. 

Summary of Invention Paragraph : 

[0006] The present invention provides a method for monitoring the interior and/or 
exterior of a vehicle having a sensor that is formed by at least one surveillance 
camera, in the case of which the interior and/or exterior is sensed by the 
surveillance camera and the image data are evaluated, wherein the evaluation is 
used to select at least one area, and wherein this area is sensed by means of a 
second alignable sensor with a restricted spatial sensing range and the data 
recorded by the second sensor are subjected to an evaluation. The present invention 
also provides a vehicle having a first sensor which is formed by at least one 
surveillance camera in the interior of the vehicle and whose field of view at least 



partially covers the interior and/or exterior of the vehicle, wherein a second 
sensor is provided which has a restricted spatial sensing range and whose alignment 
can be controlled as a function of the first sensor. 

Summary of Invention Paragraph : 

[0008] In accordance with the present invention, the vehicle interior and the 
vehicle exterior, as well, are observed by means of at least one surveillance 
camera that comprises, in a preferred embodiment, a conventional digital, 
particular CCD, camera and a, for example, spherically or parabolically convex 
mirror that is set apart from the camera and is observed, in turn, by the camera . 
Surveillance cameras in an integrated housing have also proved themselves, in 
addition. Surveillance cameras are described, for example, in PCT international 
patent publications WO 99/30197, WO 99/45422 and WO 97/43854, and are used, for 
example, for monitoring purposes and in the case of robot navigation . They 
typically produce a 360. degree, panoramic image in a way similar to a fish eye 
camera . Unlike fish eye cameras, which virtually no longer permit details to be 
recognized on the taking horizon, that is to say at the edge of its azimuthal 
taking range of max. 180. degree., surveillance cameras also reproduce details in 
the edge region of an image and thereby even permit, if appropriate, azimuthal 
taking ranges of more than 180. degree.. 

Summary of Invention Paragraph : 

[0009] Given a suitable arrangement of the surveillance camera, in particular in 
the region of the inside mirror, a very large part of the vehicle interior, and 
also of the vehicle exterior, can be sensed at once. It has also proved effective 
to arrange a convex mirror in the vehicle interior on the vehicle roof, as a result 
of which the entire hemisphere situated therebelow, that is to say virtually the 
entire vehicle interior, and also the exterior that can be sensed through the side 
window panes, can be taken. 

Summary of Invention Paragraph : 

[0010] It has proved effective, furthermore, to integrate the convex mirror or the 
camera itself in the dashboard, in particular when it is principally the front area 
of the vehicle interior and the area in front of the vehicle that are to be 
monitored. Image data recorded by the surveillance camera can be used to select an 
interesting area for a more detailed evaluation, and to make use, for a more 
detailed evaluation, of a second sensor, which is distinguished by a spatially 
restricted sensing range and is designed to be capable of alignment such that it 
can be aligned with an area classified as interesting. The data recorded by the 
second sensor are subjected to an evaluation that gives more detailed information 
relating to the selected area than is generally permitted by the surveillance 
camera alone. 

Summary of Invention Paragraph : 

[0011] The present invention creates a method that can make available reliable 
information in relevant, selected areas of the exterior or else the interior of the 
vehicle. The comprehensive recording of the image data of the surveillance camera 
renders possible a very reliable sensing and selecting of the relevant area or 
areas of particular interest, without it being possible to overlook or not consider 
individual areas that can be important for a driving decision. In order to permit a 
reliable driving decision, the method according to the present invention or the 
vehicle according to the present invention is used by selecting particularly 
relevant areas and feeding them to more detailed sensing by a second sensor, in 
particular one with special properties. The reliability of evaluation, and thus 
also the driving safety, are substantially increased thereby. 

Summary of Invention Paragraph : 

[0012] On the basis of the recorded and evaluated image data, the driver can be 
warned at an early stage, on the one hand, and on the other hand it is possible to 
effect measures to prevent accidents by means of active intervention in the 



vehicle, or else to effect measures to limit the severity of consequences of 
accidents, for example by early triggering of airbags or the like. 



Summary of Invention Paragraph : 

[0013] According to a particularly preferred embodiment of the present invention, 
the second sensor is formed from at least a digital camera, an infrared camera, a 
laser point sensor, a radar sensor or a combination thereof. As a result, the laser 
point sensor or the radar sensor, in particular, create a very reliable evaluation 
of the relative behavior of the sensed, selected area of the second sensor in 
relation to the vehicle. In particular, the relative speed or else the distance 
from the vehicle is sensed, evaluated and made available for further processing in 
the vehicle. Precisely through use as second sensors of sensor types that reliably 
permit sensing of the relative speed of objects in the sensed area, very important 
information is obtained for preventing, or limiting the consequences of, accidents, 
and this benefits the driving safety of the vehicle itself, but also of the traffic 
as a whole, in particular the safety of pedestrians. However, infrared cameras or 
digital cameras with a relatively large magnification factor, in particular with a 
zoom function, also prove to be very useful, since they additionally permit 
substantially more detailed information to be obtained, in particular under 
unfavorable situations such as fog or dusk or night, in relation to the information 
obtained by the surveillance camera . This additional information is made available 
to the vehicle per se or in combination with the data from the first sensor, and 
the vehicle is correspondingly controlled to enhance traffic safety. 

Summary of Invention Paragraph : 

[0014] The alignment of the second sensor is preferably performed on the basis of 
an automated evaluation of the image data of the first sensor, by virtue of the 
fact that the second sensor is swiveled either by motor, and thereby swivels the 
restricted sensing range onto another area of the sensing range of the surveillance 
camera, or electronically, as performed, for example, in the case of a changed 
drive or phased array radar antenna. In the case of the latter, one and the same 
radar antenna achieves a different directional characteristic by differentiated 
driving, without the need for the antenna to be swiveled mechanically or by motor 
relative to the sensor. Such an electronic alignment of the second sensor proves to 
be very advantageous, since mechanically swivelable sensors have proved to be very 
susceptible owing to the continuous shaking and vibration in vehicles. 

Summary of Invention Paragraph : 

[0015] The alignment of the second sensor is preferably carried out on the basis of 
automated evaluation of the image data, methods for the analysis of movement, 
contour and/or color having proved themselves, in particular, for evaluating image 
data. This evaluation of the image data of the first sensor results in automated 
selection of an area of particular interest which is subsequently subjected to a 
thorough more detailed observation by the second sensor. In this case, it has 
proved to be particularly effective to carry out the selection of the area of 
interest with the aid of an evaluation of movements in the image of the first 
sensor, for example by using the optical flux, and this has proved to be 
particularly effective in the case of using the present invention in conjunction 
with a device for restricting or preventing collisions with pedestrians or 
cyclists . 

Summary of Invention Paragraph : 

[0016] Since the images obtained by the at least one surveillance camera are 
greatly distorted, that is to say are present in some form of curvilinear "world 
coordinates", one or more undistorted partial images are generated therefrom by 
transforming the images of the camera into cylindrical or plane coordinates. The 
relationship between the curvilinear coordinate system of the camera images and the 
cylindrical or plane target coordinate systems is fixed by the mirror geometry and 
camera and/or by the structure of the surveillance camera . In the transformation, 
the values of brightness and, if appropriate, color of each image point of a camera 



image are assigned to a point in the cylindrical or plane coordinate system, whose 
coordinates result from trigonometric relationships, for example in the case of a 
spherical mirror. 

Summary of Invention Paragraph : 

[0017] The corresponding calculations can be carried out substantially in real time 
in a computer in the vehicle; in order to save computing power, the described 
assignment is carried out in practice, however, preferably with the aid of one or 
more transformation tables that are drawn up during a camera calibration and stored 
for the purpose of use during the camera operation in an onboard computer or a 
hard-wired electronic image rectification system. 

Summary of Invention Paragraph : 

[0018] This leads to one or more partial images of the vehicle interior in the case 
of which substantially only a one-dimensional distortion is present (in the case of 
a transformation to cylindrical coordinates) or (in the case of a transformation to 
plane coordinates) no distortion at all is present any more, and so straight lines 
are essentially reproduced as straight lines. Such images in cylindrical or plane 
coordinates can then be further processed electronically in a very simple way, in 
particular they can be evaluated very simply. This permits simple further 
processing, and thus cost-effective implementation of the present invention in a 
vehicle. In particular, the evaluation, the selection of a particularly interesting 
area for closer evaluation by the second sensor with the aid of a selection stage, 
is made substantially easier. Moreover, it is possible by this transformation to 
achieve a modularization of the monitoring system for a vehicle, and this permits a 
simple replacement of the at least one surveillance camera with subsequent 
transformation to the respective circumstances of a vehicle in conjunction with 
largely identical subsequent image processing and evaluation with selection of the 
areas of interest . It is thereby possible to lower substantially the costs for such 
systems for monitoring the interior and exterior of a vehicle, and thereby to raise 
the acceptance to the user without appreciable loss in the reliability of 
evaluation . 

Summary of Invention Paragraph : 

[0019] The present invention also relates to a vehicle having a first sensor that 
is formed by at least one surveillance camera in the interior of the vehicle, whose 
field of view at least partially covers the interior and/or exterior of the 
vehicle. The first sensor is assigned a second sensor that has a restricted spatial 
sensing range of which the alignment can be controlled as a function of the first 
sensor. In this case, the alignment is preferably performed via a control unit that 
can be controlled on the basis of automated evaluation of the image data of the 
first sensor, which is preferably carried out by an image evaluation unit, such 
that a selected region, classified as particularly interesting or relevant, of the 
visual range of the first sensor is specifically sensed by the second sensor and 
thoroughly evaluated. In this case, the alignment of the second sensor is performed 
by means of a control unit that swivels the second sensor preferably by motor, or 
adapts its alignment correspondingly in an electronic way. The result of this is a 
preferably automated recording of the relevant information from the exterior or 
interior of the vehicle with the aid of the selection by a selection stage in 
conjunction with a corresponding control unit, which is assigned to the first 
sensor, and a very reliable mode of operation is thereby provided for the method 
for monitoring the interior and exterior of a vehicle. 

Summary of Invention Paragraph : 

[0020] It has proved to be especially advantageous either to use the first and/or 
the second sensor per se in each case, or to use them in common as a stereoscopic 
sensing system for the interior and/or exterior of the vehicle. Consequently, the 
recorded information of each sensor which is formed at least by two individual 
sensors is, for example, formed by two surveillance cameras or from two digital 
cameras, two infrared cameras, two laser point sensors or two radar sensors or a 



combination of two such individual sensors in such a way that a stereoscopic 
evaluation of the sensing range is possible with two individual sensors. In the 
course of this stereoscopic evaluation, it is possible, in particular, to record 
and evaluate information relating to the depth graduation of the objects in the 
sensing range and, in particular, information relating to the distance or else to 
the change in distance, that is to say the relative speed. This stereoscopic 
information permits warning functions to be activated very specifically, or 
activation of defensive strategies for preventing or limiting the effects of 
accidents by early activation of defensive measures, so-called precrash measures, 
or else with regard to intervention in the driving behavior of the vehicle, for 
example by means of independent, autonomous braking or evasion of the vehicle. It 
is thereby particularly the information relating to the spatial breakdown of the 
exterior of a vehicle, particularly in the front region, that forms the basis of 
the control. 

Summary of Invention Paragraph : 

[0021] The use of infrared cameras, laser point sensors and/or radar sensors 
results in a very reliable way in expansion of the information content of the 
sensible surroundings in the interior and/or exterior of the vehicle via the 
information content of a camera that substantially operates exclusively in the 
visible frequency range. A substantially differentiated representation of the 
information relating to the surrounding area is thereby rendered possible and made 
available for later evaluation of the vehicle. 

Summary of Invention Paragraph : 

[0023] It has proved to be particularly effective to arrange the first sensor, 
which includes one or two or also more surveillance cameras, in the roof area, 
particularly in the region of the inside mirror, and this results in a very 
advantageously structured sensing range of the first sensor. In particular, the 
lateral area and the front area of the vehicle can be very effectively sensed 
through the window panes of a vehicle here, but also so can the interior, in 
particular the area of the front seats, and can therefore be evaluated very easily 
with regard to the selected areas. 

Summary of Invention Paragraph : 

[0026] By contrast with the case of other conventional optical sensor systems for 
vehicles, the combination of the at least one surveillance camera in conjunction 
with the second alignable sensor with a restricted field of view permits a 
reduction in the number of the cameras required for carrying out the multiplicity 
of possible tasks in recording information from the interior and/or from the 
exterior . 

Summary of Invention Paragraph : 

[0027] In addition to said possible application of the present invention in 
conjunction with assistant systems for the detection of traffic lights, detection 
of traffic signs, methods for tracking traffic jams, lane detection, detection of 
the right/left situations, object detection in the near field, such as cyclists, 
for example, or sensing and evaluating the situation at an intersection, it is also 
possible to implement other applications such as an interior monitoring for 
antitheft security or for documenting traffic situations, particularly in 
connection with accidents. The applications and image evaluation systems that come 
to be applied in connection with the present invention do not require calibrated 
systems; it is also possible to use uncalibrated systems. Again, it is possible to 
apply the present invention in other vehicles, which are not automobiles, 
particularly in aircraft or ships, for example for monitoring tasks. 

Brief Description of Drawings Paragraph : 

[0030] FIG. 2 shows a sketch of an image of a surveillance camera ; and 



Brief Description of Drawings Paragraph : 



[0031] FIG. 3 shows a rectified partial image in accordance with FIG. 2. 



Detail Description Paragraph : 

[0032] An exemplary design of an arrangement according to the present invention for 
monitoring the interior and/or exterior of a vehicle having two surveillance 
cameras is illustrated in FIG. 1. 

Detail Description Paragraph : 

[0033] The first surveillance camera comprises a spherically or parabolically 
convex mirror 1 and a digital camera 2 that constitutes a CCD camera . A second 
surveillance camera is constructed correspondingly from a second mirror 3, which is 
designed as a spherically or parabolically convex mirror 3, and from a CCD camera 
4. The two mirrors 1, 3 are arranged on the roof of the vehicle in the interior. 
The two cameras 2, 4 are arranged below the two mirrors 1, 3 and have the two 
mirrors 1, 3 in their field of view, in particular comprising the essential field 
of view of the cameras 2, 4. Such surveillance cameras are described in, for 
example, said international patent documents WO 99/30197, WO 99/45422 and WO 
97/43854. The convex mirror 1 is fitted in this example on the roof above the area 
between the front seats, the reflecting surface pointing downward, and the assigned 
camera 2 being fastened between the two front seats with sight line upward in the 
direction of the mirror 1. The second convex mirror 3 is arranged in the middle of 
the vehicle roof. Its reflecting surface likewise points downward. The camera 4 is 
arranged below the mirror 3 in the footwell of the vehicle in the rear compartment 
in the region of the transmission tunnel such that it is aligned with the mirror 3. 



Detail Description Paragraph : 

[0034] In the case of this arrangement, the camera 2 or 4 sees in the assigned 
convex mirror 1, 3 an image of the hemisphere below the roof of the vehicle as 
illustrated schematically by way of example in FIG . 2. Here, with the exception of 
a mechanically or electronically masked central region in which it would image 
itself, the image shows the hemisphere named above. As may be gathered from FIG. 2, 
the camera senses not only the interior with the seats and the vehicle occupants, 
it is also capable of sensing the area outside through the windscreen, details of 
the exterior not having been illustrated in FIG. 2, in order to improve 
comprehensibility . The illustration was limited to reproducing the window panes, in 
order to improve clarity, and so the exterior is not reproduced. 

Detail Description Paragraph : 

[0035] The digital image data supplied by the cameras 2, 4 are strongly distorted, 
since they image the surroundings in spherical or some other curvilinear 
coordinates, depending on the shape of the mirror. Each image of the cameras 2, 4 
is fed to a rectifying unit 5 in which one or more parts of the image are 
transformed to plane coordinates. An exemplary transformed image of the driver side 
is illustrated in FIG. 3. The image illustrated shows a relatively undistorted 
image in which straight lines are also reproduced as substantially straight lines. 

Detail Description Paragraph : 

[0036] The transformed image data are fed to a selection stage 6 which is now 
enabled in a simple way on the basis of the transformed, rectified image data to 
select interesting areas of the image recorded by the cameras 2, 4 by analyzing 
contours, colors and movements, for example using the concept of optical flux. 

Detail Description Paragraph : 

[0037] If the arrangement is used to monitor the interior and/or exterior in the 
case of a pedestrian monitoring unit, it is preferred to use a selection stage with 
movement analysis, while given an application as a traffic lights or traffic signs 
assistant it is analysis by means of a contour and/or color that are/is applied. 
If, in an automated process, the selection stage determines an area as particularly 
relevant, and thereby selects this area, an item of information representing this 



selected area is reported by the selection stage 6 to the control unit 7 which then 
uses the alignment unit 8 to swivel the second sensor 9, which includes a CCD 
camera with zoom function, to the effect that the field of view of the second 
sensor 9 covers this selected area. Here, the magnification factor (zoom factor) of 
the second sensor 9 is set by the control unit 7 such that the objects in the 
selected area can be sensed in detail. The zoom factor is selected here in 
accordance with the distance, determined by a stereoscopic measurement, of the 
selected area or of the objects in the selected area. 

Detail Description Paragraph : 

[0038] The stereoscopic measurement is performed in this case via the two 
surveillance cameras 1, 2/3, 4, which together form a stereoscopic surveillance 
camera. The stereoscopic evaluation is performed here by the selection stage 6, 
which makes available the distance information of the control unit 7, which 
consequently controls the zoom of the second sensor 9. 

Detail Description Paragraph : 

[0039] The image data recorded by the second sensor 9 and the two surveillance 
cameras 1, 2/3, 4 are fed to an image evaluation unit 10 that permits an overall 
evaluation of the image data of all the sensing systems, and thus of the two 
sensors, that is to say the first surveillance camera 1, 2, the second surveillance 
camera 3, 4 and the zoom camera 9. It is possible in the course of the overall 
evaluation in particular to resolve instances of ambiguity and/or to permit a very 
specific evaluation of the spatial subdivision of the sensed exterior and/or 
interior. As a result, it is possible in particular to determine distances and/or 
positions of objects individually sensed. Moreover, relative speeds of sensed 
objects can also be calculated in relation to the vehicle or to the sensor 
arrangement. It is possible precisely by means of the exemplary overall evaluation 
of all image information to obtain information that is very informative and 
reliable for the purpose of constructing the exterior and/or the interior of the 
vehicle. With the aid of this secure and reliable information, other components of 
a vehicle can make necessary measures available, for example warnings to the driver 
or codriver or measures for further information for the driver and/or co-driver, 
and/or initiate measures for reducing effects of accidents such as, for example, 
early inflation of airbags or early inclining of the engine hood before a 
pedestrian impact on the vehicle, or measures for automatically braking or 
accelerating a vehicle or avoiding contact by it. For this purpose, the required 
information of the image evaluation unit is made available to these other 
components of the vehicle via an interface 11. 

Detail Description Paragraph : 

[0040] The method according to the present invention is suitable in a particularly 
advantageous way for use in vehicles, particularly in conjunction with a device for 
protection against theft, or with a device for transmitting image data. In 
particular, the image data are transmitted via a mobile radio telephone to persons, 
for example an owner of a motor vehicle, as soon as the alarm system or the 
antitheft device is activated. 

Detail Description Paragraph : 

[0041] Moreover, the present invention is particularly suitable for cooperating 
with a recording system that senses and stores the driving situation at the same 
time as an accident both in the interior and in the exterior of the vehicle such 
that a later analysis of the accident is permitted. Owing to the cooperation of the 
two sensors once as panoramic sensor (surveillance cameras ) , and once as selected 
sensor (second sensor) for particularly relevant areas, it proves to be very 
helpful that precisely the information that is particularly important for an 
accident situation, for example the overall view, but also special areas can be 
sensed and documented specifically. 



CLAIMS: 



I. A method for monitoring a space of a vehicle comprising: sensing the space using 
a first sensor including at least one surveillance camera so as to produce an image 
data; performing an evaluation of the image data; selecting at least one area of 
the space using the evaluation of the image data; sensing the at least one area 
using a second alignable sensor having a restricted spatial sensing range so as to 
produce a second data; and evaluating the second data. 

4 . The method as recited in claim 3 wherein the aligning of the second sensor is 
performed using a control unit based on the evaluation of the image data. 

7. The method as recited in claim 4 wherein the evaluation of the image data is 
performed automatically using the control unit. 

8. The method as recited in claim 7 wherein the evaluation of the image data 
includes analyzing at least one of a movement, a contour and a color. 

10. The method as recited in claim 1 wherein the image data has curvilinear 
coordinates and further comprising transforming the image data so as to have 
cylindrical or plane coordinates before the evaluation of the image data is 
performed. 

II. A vehicle comprising: a first sensor including at least one surveillance camera 
disposed in an interior of the vehicle, the first sensor having a field of view 
covering at least one of a portion of the interior and a portion of an exterior of 
the vehicle; and a second alignable sensor having a restricted spatial sensing 
range, an alignment of the second sensor being controllable using the first sensor. 

12. The vehicle as recited in claim 111 wherein the first sensor produces image 
data and further comprising: an electric or motor drive; a control unit; and a 
selection stage for performing an automated evaluation of the image data, wherein 
the drive and the control unit are configured to align the second sensor using the 
automated evaluation. 

14. The vehicle as recited in claim 11 wherein the first sensor includes two 
surveillance cameras jointly forming a stereo image camera . 

15. The vehicle as recited in claim 11 wherein the second sensor includes at least 
one of a digital camera, an infrared camera, a laser point sensor and a radar 
sensor. 



Previous Doc 



Next Doc 



Go to Doc# 



